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ABSTRACT 


This  report  presents  the  results  of  a  study  to  develop  basic  relation¬ 
ships  capable  of  determining  and  predicting  the  acquisition  costs  attributable 
to  equipment  reliability.  The  study  is  based  on  data  derived  during  the  de¬ 
velopment  phase  of  equipment  acquisition.  The  relationships,  including  in¬ 
cremental  reliability  gain  related  to  incremental  reliability  cost,  were  de¬ 
veloped  using  data  from  two  manufacturers  on  ten  equipments  covering  three 
reliability  elements.  The  equipments  cover  a  span  of  end  use  airborne  and 
space  environments .  The  reliability  elements  analyzed  in  the  study  are  the 
reliability  design  program  (includes  prediction,  failure  mode  and  effect 
analysis,  and  design  reviews);  the  reliability  parts  program  (includes  parts 
screening  specification,  parts  standardization  and  control,  and  vendor  con¬ 
trol);  and  the  reliability  testing  program  (includes  evaluation  testing,  equip¬ 
ment  environmental  screening,  and  reliability  demonstration  testing). 

To  develop  the  relationships,  two  linear  models  were  hypothesized. 

The  first  model  relates  resultant  equipment  reliability,  the  reliability  costs 
and  the  equipment  complexity.  The  second  model  relates  incremental  re¬ 
liability  element  gain  to  reliability  element  cost. 

Detailed  discussions  of  data  collection  and  analysis  efforts  along  with 
step-by-step  procedures  for  using  the  modeling  results  are  presented  in  the 
report.  Constraints  and  precautions  to  be  applied  in  using  the  equatlonal  re¬ 
lationships  are  also  included . 
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EVALUATION 


1.  The  objectives  of  this  study  were  to: 

a.  Develop  relationships  between  resultant  reliability  and  reliabil¬ 
ity  development  cost  for  avionic  equipment. 

b.  Determine  sensitivity  relationships  and  trade-offs  among  the 
elements  comprising  a  reliability  program  to  resulting  equipment  reli¬ 
ability. 

2.  Both  objectives  were  met.  Sufficient  valid  data  were  available  to 
provide  the  desired  relationships  with  a  high  order  of  statistical 
significance.  The  data  were  such  that  two  types  of  analyses  were  possible: 

a.  A  macroscopic  analysis  which  provided  the  various  cost  estimating, 
reliability  estimating,  and  rudimentary  trade-off  relationships  required. 

b.  A  microscopic  analysis  which  had  as  its  objective  the  development 
of  "Reliability  Rain"  relationships  relative  to  various  elements  and 
subelements  of  a  reliability  program  based  on  their  cost  and  impact.  The 
analysis  had  as  its  purpose  the  determination  of  the  optimal  allocation  of 
resources  to  tho  various  elements  and  subelements  of  a  reliability  program 
assuming  "failure  analysis  and  fix"  feedback  loops  exist  during  the 
development /test  program. 

3.  The  results  of  the  program  will  be  used  in  trade-off  and  life  cycle 
cost  analyses  providing  a  hereto foie  missing  link,  ».  relationship  between 
reliability  development  cost  and  resulting  reliability!  the  results  can' 
also  be  used  as  a  foundation  to  structure  the  bptimtih  eize  and  mix  of  a 
reliability  program  in  a  development  environment  similar  to  the  one  from 
which  the  data  eminated. 

4.  This  effort,  supports  RADC  Technology  Plan  TPO-13,  Reliability, 
paragraph  3.13.2.2,  Reliability  Management. 

M.  H  / 

.TElyMF  KIT  ON 

Rel,  G  Maint.  Engineering  Section 
Reliability  Branch 
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SECTION  I 


INTRODUCTION 


1 .  BACKGROUND 

The  Air  Force  has  indicated  great  interest  in  equipment  reliability  and 
equipment  reliability  costs .  It  has  been  apparent  lor  many  years  that,  for 
an  additional  cost  increment  during  development  aimed  at  making  an  equip¬ 
ment  more  reliable,  many  times  that  cost  would  be  recoverable  over  the 
life  of  the  equipment  due  to  its  improved  reliability.  While  a  number  of 
studies  have  qualitatively  addressed  the  cost/reliability  problem  or  quanti¬ 
tatively  addressed  the  cost-of-failure  problem,  only  a  few  studies  have 
attacked  the  problem  of  quantification  of  incremental  development  cost  as  a 
function  of  incremental  reliability  improvement.  This  study  has  used  histor¬ 
ical  failure  rate  data  along  with  historical  cost  data  to  truly  quantify  im¬ 
provement  in  equipment  reliability  as  a  function  of  incremental  reliability 
element  cost. 

2 .  STUDY  OBJECTIVES 

The  objectives  of  this  study  were  to  develop  relationships  capable  of 
determining  and  predicting  the  costs  attributable  to  reliability  during  the 
development  phase  of  electronic  equipment  acquisition.  Further,  basic  re¬ 
lationships  were  to  be  developed  equating  reliability  increments  to  incre¬ 
ments  in  development  cost.  Specifically,  the  key  objectives  identified  at  the 
outset  of  the  study  were  to: 

•  Develop  a  relationship  between  the  equipment  reliability  and  the 
total  reliability  development  cost. 

•  Develop  a  relationship  between  reliability  element  costs  and  the 
equipment  reliability. 

•  Develop  relationships  between  reliability  improvement  and  re¬ 
liability  cost  for  each  of  the  reliability  elements  or  groups  of 
reliability  elements . 


3. 


RELIABILITY  ELEMENTS 


The  study  dealt  with  three  reliability  elements  as  they  relate  to  ten 
equipments  from  two  manufacturers.  These  elements  are: 

•  Reliability  Design  Program  -  including  prediction,  failure  mode 
and  effects  analysis  (FMEA),  and  design  reviews. 

•  Reliability  Parts  Program  -  including  parts  screening  specifica¬ 
tion,  parts  standardization  and  control,  ar.d  vendor  control. 

•  Reliability  Testing  Program  -  including  evaluation  testing,  equip¬ 
ment  environmental  screening,  and  reliability  demonstration 
testing. 

These  elements  and  their  composition  are  described  in  Section  II. 

4 .  DATA  SOURCES 

The  equipments  and  their  respective  manufacturers  were  the  magnetic 
tape  transport  from  the  Test  Equipment  Division  of  Honeywell,  Inc.,  and 
four  radar  equipments,  three  digital  equipments,  a  countermeasures  equip¬ 
ment,  and  a  military  television  equipment  from  the  Aerospace  Electronic 
Systems  Department  of  the  General  Electric  Company. 

This  study  has  also  considered  the  results  of  two  other  studies  done 
previously  and  independently.  They  were  a  cost  analysis  effort  by  the  Naval 
Weapons  Center,  China  Lake,  California  and  the  results  of  the  work  reported 
by  Avco  Corporation,  Wilmington,  Massachusetts  (Ref.  1). 

5.  GENERAL  APPROACH 

The  approach  to  the  study  was  to  hypothesize  linear  models  (linear  in 
coefficients  but  not  necessarily  linear  in  the  variables)  for  developing  re¬ 
lationships  between  equipment  reliability  and  reliability  cost;  collect  relia¬ 
bility  cost  data,  reliability  data  (based  on  failure  rate  history)  and  normali¬ 
zation  data  on  the  various  equipments;  correlate  the  data;  synthesize  the  data 
to  the  models  and  iterate  the  models  to  satisfaction.  Ail  of  the  data  analyses 
were  performed  using  time-sharing  computer  programs  developed  by  the 
Information  Services  Business  Department  of  the  General  Electric  Company 
in  cooperation  with  the  General  Electric  Corporate  Research  and  Develop¬ 
ment  Center. 
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6. 


REPORT  OVERVIEW 


Section  II  describes  the  makeup  of  the  reliability  elements  studied  and 
details  the  efforts  associated  with  collection  and  evaluation  of  the  normali¬ 
zation,  cost,  and  reliability  data.  Section  III  contains  the  analytical 
approach  used  in  the  study,  while  Section  IV  details  the  results  obtained. 
Section  V  presents  the  conclusions  and  recommendations  relating  to  this 
studji  and  possible  future  efforts. 
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SECTION  II 


COLLECTION  AND  EVALUATION  OF  DATA 


1.  DESCRIPTION  OF  RELIABILITY  ELEMENTS 

This  study  investigated  the  three  reliability  elements  (reliability  de¬ 
sign  program,  reliability  parts  program,  and  reliability  test  program)  that 
are  considered  basic  to  the  achievement  of  equipment  reliability.  Each  of 
these  elements  is  further  broken  down  into  three  subelements  or  tasks. 

These  elements  and  their  component  subelements  are  discussed  below. 

a.  Reliability  Design  Program 

The  reliability  design  program  consists  of  reliability  prediction, 
failure  mode  and  effects  analyses,  and  design  reviews.  Each  of  these  tasks 
is  interrelated  to  every  other  task.  For  instance,  the  prediction  is  useful 
with  the  FMEA  and  both  are  used  in  design  reviews .  These  taskB  of  the  re¬ 
liability  design  program  are  a  prerequisite  for  the  reliability  parts  program. 

(1)  Reliability  Prediction 

Reliability  Prediction  is  used  as  a  tool  to  ascertain  the 
achievable  MTBF  for  the  design  at  various  stages  in  the  equipment  develop¬ 
ment.  The  technique  involves  the  use  of  stress  ratios  for  each  component 
part  in  the  intended  use  environment  to  establish  failure  rates  as  described 
in  MIL-HDBK-217A  or  the  RADC  Notebook.  A  prediction  is  derived  for  each 
assembly  and  inserted  into  a  reliability  model  for  the  equipment  to  determine 
compatibility  with  the  contractually  required  MTBF. 

(2)  Failure  Modes  and  Effects  Analyses 

Failure  Modes  and  Effects  Analyses  (FMEA's)  are  conducted 
during  the  design  program  to  identify  the  effects  of  potential  failure  modes 
and  potential  system  weaknesses  on  system  performance.  A  functional  flow 
diagram  is  developed  to  depict  the  series  elements  for  the  operational  modes. 
Critical  elements  are  identified  for  more  detailed  study,  sometimes  to  the 
component  level.  Through  these  analyses,  reliability  improvement  emphasis 
is  directed  to  design  areas  critical  to  mission  success. 
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(3)  Design  Review 

Design  reviews  are  conducted  with  Engineering  and  other  re¬ 
lated  functions  for  assessment  of  the  reliability  design  status  and  to  identify 
and  correct  potential  problems  with  the  design.  The  results  of  reliability 
predictions  and  FMEA's  are  used  as  reliability  tools  during  these  reviews 
and  to  provide  an  early  evaluation  of  alternative  design  proposals.  Formal 
reviews  are  planned  and  held  to  a  schedule  based  on  the  design  status.  Many 
informal  reviews  are  also  held  between  design  personnel  and  a  reliability 
engineer  on  an  as-needed  basis  to  assure  the  design  will  meet  the  contractual 
reliability  requirement. 

b.  Reliability  Parts  Program 

The  second  reliability  element  evaluated  during  the  study  was  the 
reliability  parts  program,  which  consists  of  parts  standardization  and  con¬ 
trol,  parts  screening  specification,  and  vendor  control.  AESD  experience 
has  shown  that  the  reliability  tasks  making  up  the  parts  program  yield  parts 
with  lower  failure  rates  at  lower  cost.  Therefore,  it  is  considered  essential 
that  a  comprehensive  parts  program  be  imposed  as  a  prerequisite  to  tho 
third  element,  the  reliability  test  program. 

(1)  Parts  Standardization 

Parts  standardization  is  essential  to  minimizing  the  number 
of  part  types  used  in  an  equipment.  Through  parts  standardization,  relia¬ 
bility  efforts  are  dirocted  to  the  control  and  screening  of  a  minimum  number 
of  different  part  types  in  order  to  maximize  the  use  of  parts  with  known  re¬ 
liable  performance.  Additionally,  control  of  parts  for  reliable  application 
can  be  more  rigidly  maintained  during  the  design  phase.  For  example,  a 
typical  commercial  equipment  and  a  typical  military  equipment  were  com¬ 
pared.  No  parts  standardization  program  was  instituted  for  the  commercial 
equipment;  as  a  result,  70  percent  more  part  types  were  required  than  for  the 
military  equipment  of  approximately  the  same  parts  complexity.  Standardi¬ 
zation  like  this  offers  three  significant  advantages:  fewer  part  type  drawings, 
fewer  vendors,  and  lower  part  costs  owing  to  price  breaks  on  higher  pro¬ 
curement  quantities . 

(2)  Parts  Screening  Specification 

While  parts  standardization  efforts  are  directed  to  controlling 
part  types  in  an  equipment,  parts  screening  efforts  are  directed  at  con¬ 
trolling  the  reliability  of  parts.  These  efforts  include  the  investigation, 
preparation  and  negotiation  of  screening  specifications  to  be  applied  to  the 
component  parts.  These  specifications  include  provision  for  operating 
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burn-in,  monitoring  of  critical  parameters  during  burn-in,  in-line  inspec¬ 
tions,  nondestructive  environmental  tests  and  providing  data.  The  effective¬ 
ness  of  the  parts  screening  specification  effort  has  been  assessed  at  AESD 
through  analysis  of  test  results  on  incoming  parts .  Nonscreened  parts  have 
been  found  to  exhibit  a  defect  rate  approximately  four  times  greater  than  for 
screened  parts .  * 

(3)  Vendor  Control 

Vendor  control  is  the  effort  exerted  to  assure  the  attainment 
of  reliable  specialty  items  and  subcontract  hardware.  This  effort,  requires 
the  same  type  of  investigation,  preparation  and  negotiation  of  specifications 
as  parts  screening  but  at  a  higher  level  of  assembly.  These  vendors  are 
monitored  and  controlled  in  the  same  way  as  component  vendors. 

c .  Reliability  Test  Program 

The  third  reliability  element  considered  is  the  reliability  test  pro¬ 
gram.  This  element  is  comprised  of  three  reliability  tasks,  namely,  equip¬ 
ment  screening,  evaluation  testing,  and  qualification/demonstration  testing. 

(1)  Environmental  Screening 

Environmental  screening  at  the  equipment  level,  similar  to  the 
environmental  test  profiles  specified  in  MIL-STD-781,  is  conducted  to  re¬ 
move  infant  or  early  mortality  failures  from  the  equipment.  The  failures 
removed  normally  result  from  poor  workmanship  by  component  part  sup¬ 
pliers  and  equipment  manufacturers  which  is  not  detected  during  the  normal 
factory  in-process  inspection  and  acceptance  testing.  The  screening  te3t 
also  determines  if  the  equipment  is  capable  of  meeting  its  performance  param¬ 
eters  when  subjected  to  the  screening  environment.  The  screening  environ¬ 
ment  is  generally  compatible  with  the  intended  field  use  environments.  En¬ 
vironmental  screening  tests  are  not  intended  to  improve  the  reliability  of  the 
equipment  in  the  same  manner  as  reliability  evaluation  tests.  For  these 
reasons,  the  cost  for  planning  and  implementing  environmental  screening  has 
been  normalized  to  the  average  cost  to  screen  one  equipment. 


Although  the  cost  to  screen  the  part  is  not  included  in  this  task  for  pur¬ 
poses  of  this  study,  it  is  worth  noting  that  AESD  has  experienced  an  in¬ 
crease  of  approximately  one  dollar  per  part  on  the  average  for  screening 
in  production. 
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(2)  Evaluation  Testing 

Evaluation  tests  on  initial  equipments  manufactured  are  con¬ 
ducted  for  extended  periods  to  uncover  design  and  manufacturing  deficiencies 
that  cannot  be  detected  during  the  breadboard  design  phase.  These  tests  are 
conducted  at  the  equipment  level  to  an  environment  of  temperature  cycling 
and  periodic  vibration  patterned  from  MIL-STD-781.  As  test  time  is  accum¬ 
ulated  and  equipment  failures  are  experienced,  each  failure  is  carefully 
analyzed  and  cause  of  failure  is  determined.  Corrective  action  is  imple¬ 
mented  where  necessary  and  the  effectiveness  of  the  action  is  assessed  dur¬ 
ing  the  continuing  test.  The  costs  of  this  effort  exclude  the  material  and 
labor  to  build  the  equipment(s)  but  Include  the  test  costs  along  with  analyses 
and  corrective  actions . 

Considerable  evidence  (see  Appendix  B)  indicates  that  equip¬ 
ment  reliability  grows  exponentially  with  accumulated  test  time  during  a 
properly  conducted  evaluation  test. 

(3)  Demonstration  Testing 

During  the  reliability  demonstration  test  phase,  the  equipment 
is  exposed  to  a  MIL-STD-781  environment  to  measure  the  MTBF  achieved 
on  the  equipment  as  a  result  of  the  reliability  elements  imposed.  Reliability 
demonstration  impacts  the  reliability  of  the  final  product  in  two  significant 
areas:  (1)  it  provides  an  incentive  to  the  equipment  manufacturer  to  properly 
conduct  the  RDT&E  Reliability  Program  since  failure  to  meet  the  demon¬ 
stration  criteria  usually  carries  3evere  monetary  penalties  to  the  producer, 
and  (2)  it  provides  for  an  assessment  of  effectiveness  of  corrective  action 
prior  to  commencement  of  production.  The  costs  of  this  effort  exclude  the 
material  and  labor  to  build  the  equipment(s)  but  include  the  test  costs  along 
with  analyses  and  corrective  actions . 

For  the  purpose  of  this  study,  the  MTBF  measured  at  the  end 
of  reliability  demonstration  testing  i3  considered  to  be  the  reliability 
achieved  as  a  result  of  the  RDT&E  reliability  elements  imposed. 

2 .  DATA  COLLECTION 

Following  definitization  of  the  reliability  elements,  the  data  collection 
effort  was  carried  out.  This  effort  was  divided  into  three  categories: 

(1)  equipment  characterization  data,  (2)  reliability  cost  data,  and 
(3)  reliability  data. 
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a.  Equipment  Characterization  Data 

The  equipment  characterization  data  is  that  data  that  could  be  used 
to  assess  distinct  differences  from  one  equipment  to  another.  This  data  was 
reasonably  easily  defined  and  fairly  straightforward  to  collect  from  histori¬ 
cal  files.  The  data  is  summarized  in  Table  I. 

b.  Reliability  Cost  Data 

The  reliability  cost  data  represents  the  effort  expended  during 
equipment  development  to  perform  the  various  reliability  tasks.  The  costs 
were  converted  to  mandays  using  the  average  AESD  manpower  labor  rates 
applicable  during  the  time  span  of  development  for  each  particular  equip¬ 
ment.  The  time  spans  covered  by  the  equipment  in  the  study  are  shown  in 
Figure  1. 

(1)  Normalization 

It  was  realized  at  the  outset  that  the  equipments  being  con¬ 
sidered  covered  a  broad  range  of  use  environments,  contract  reliability  re¬ 
quirements,  types  of  designs  (i.e.,  digital  vs  analog  and  new  vs  modified), 
number  of  development  equipments,  part  complexities,  degrees  of  part 
standardization  and  time  era  when  developed.  Consequently,  it  was  recog¬ 
nized  that  it  would  be  statistically  desirable  to  perform  some  data  normali¬ 
zation  to  reduce  the  number  of  independent  variables  and  maximize  the  de¬ 
grees  of  freedom  in  the  analysis.  Review  of  the  data  on  the  ten  equipments 
led  to  a  decision  to  normalize  all  equipments  to  an  airborne,  analog,  new 
design  configuration. 

(a)  Space  to  Airborne  Normalization 

A  space  environment  is  less  severe  than  an  airborne  en¬ 
vironment  but  the  equipment  goes  through  a  more  severe  missile  environ¬ 
ment  to  reach  space.  Therefore,  since  this  study  only  included  one  space 
equipment,  it  was  assumed  that  on  the  average,  the  space  equipment  would 
be  comparable  to  an  airborne  equipment  with  respect  to  use  environment. 

(b)  Modified-Design-To-New-Design  Normalization 

Since  the  study  includes  only  one  equipment  with  a  modi¬ 
fied  design,  the  historical  data  on  that  equipment  was  reviewed  to  determine 
the  portion  of  the  equipment  (number  and  complexity  of  new  circuits)  that  was 
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TABLE  I.  EQUIPMENT  CHARACTERIZATION  DATA  BY  EQUIPMENT 


modified.  That  is,  on  this  particular  radar,  the  quantity  of  parts  that  were 
new  to  the  design  was  2,632  out  of  the  total  quantity  of  parts  of  11,545. 
Therefore,  the  quantity  of  parts  for  this  equipment  was  changed  to  2,632  to 
correspond  to  the  portion  of  the  design  that  was  new. 

(c)  Digital  to  Analog  Normalization 

In  general ,  a  digital  equipment  of  the  same  complexity  a.s 
an  analog  equipment  has  fewer  different  types  of  parts  and  fewer  different 
circuits.  This  means  design  and  part  costs  will  differ  even  though  complex¬ 
ity  is  the  same.  It  was  decided  early  in  the  study  to  modify  cost.  In  an 
effort  to  obtain  a  better  assessment  of  this  cost  modifier,  the  relationship 
between  number  of  electrical  parts  and  number  of  part  drawings  was 
examined  for  the  ten  equipments.  This  relationship  is  shown  in  Figure  2. 
Since  parts  and  design  costs  are  directly  related  to  complexity,  a  cost  modi¬ 
fier  was  obtained  by  dividing  the  relationship  for  analog  equipments  by  the 
relationship  for  digital  equipments.  It  should  be  kept  in  mind  that  this  re¬ 
lationship  is  based  on  historical  data  that  resulted  from  part  drawing  stand¬ 
ardization  practices  developed  at  AESD  (Ref.  2).  Anyone  using  these  rela¬ 
tionships  should  consult  Reference  2  to  assure  consistency  of  practices. 

The  cost  modifiers,  as  calculated  from  the  Figure  2 
relationships  for  the  three  digital  equipments  in  this  study,  are  summarized 
in  Table  II.  These  modifiers  were  applied  to  each  respective  digital  equip¬ 
ment  for  the  design  discipline  cost  and  the  parts  discipline  cost  only.  The 
test  discipline  cost  was  left  unchanged  since  it  was  assumed  that  the  addition¬ 
al  effort  in  design  and  parts  would  fully  adjust  off  the  digital  to  analog  con¬ 
version.  The  reliability  element  cost  data  with  modifiers  applied  is  sum¬ 
marized  in  Table  III. 

c.  Reliability  Data 
(1)  General 

In  addition  to  the  collection  of  the  equipment  characteristic 
data  and  cost  data  (which  is  used  to  develop  the  gross  prediction  equations) , 
reliability  data  was  developed  on  the  basis  of  Engineering  assessment  of 
equipment  failure  rate  history  and  used  in  conjunction  with  the  equipment 
characterization  data  and  cost  data  to  develop  the  allocation  equations. 

These  allocation  equations  provide  a  quantification  of  the  reliability  contribu¬ 
tion  of  each  of  the  reliability  elements.  Both  the  gross  prediction  equations 
and  the  allocation  equations  are  detailed  in  Section  IV  of  this  report. 
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TABLE  n.  DIGITAL-TO-ANALOG  COST  MODIFIERS 


TABLE  HI.  RELIABILITY  ELEMENT  COST  DATA  (MANDAYS) 


The  Engineering  assessment  of  equipment  failure  rate  history 
on  Program  E  is  included  here  as  an  example  of  how  the  incremental  relia¬ 
bility  data  was  developed.  This  equipment  included  the  following  reliaDiliiy 
elements  which  are  typical  of  the  ten  equipments  in  the  data  base. 


•  Reliability  Design  Program  (periodic  reliability  esti  ¬ 
mates,  FMEA  and  design  reviews. 

•  Reliability  Parts  Program  (develop  and  implement  re 
quired  parts  screening  specifications,  parts  drawing 
control,  part  drawing  control,  part  selection,  parts 
evaluation/qualification  test  and  parts  application 
criteria) . 

•  Reliability  Test  Program  (equipment  environmental 
screening,  reliability  evaluation  tests,  reliability 
demonstration  test) . 


The  reliability  program  elements  can  be  shown  schematically 
as  a  series  of  events  (Figure  3)  with  the  feedback  loops  which  are  inherent  in 
a  well-integrated  reliability  program,  such  as  that  conducted  on  Program  E. 


The  values  of0  0  cj,  0  ,  ^  and  0  r  in  Figure  3  were  deter¬ 
mined  through  a  retrospect  investigation  of  the  reliability  growth  processes 
and  effective  achieved  MTBF  (0 )  levels  and  milestones  associated  with  each 
reliability  program  element.  Upon  establishment  of  the  various  0's,  and 
"inputs"  and  "outputs"  of  each  element,  the  reliability  gain  (MTBF  gain)  for 
each  element  was  calculated  as 


where 


r2  ;  etc . 

ed 


gain  in  MTBF  due  to  reliability  design  program 

(  0  Jo.  ) 

uut  in 

gain  in  MTBF  due  to  reliability  parts  program 
gain  in  MTBF  due  to  reliability  test  program 
gain  in  MTBF  due  to  total  reliability  program 


0.  =  effective  achieved  MTBF  without  benefit  of 
‘  reliability  program 
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>;  ^  =  effective  achieved  MTBF  resulting  from  relia¬ 
bility  design  program 

0  =  effective  achieved  MTBF  resulting  from  relia- 

^  bility  design  and  reliability  parts  programs 

e  t  =  effective  achieved  MTBF  resulting  from  relia¬ 
bility  design  plus  reliability  parts  plus  reliability 
test  programs 

6  R  =  resultant  equipment  MTBF  (eR  =6 
(2)  MTBF  Contribution  of  a  Reliability  Test  Program 

The  first  pair  of  0's  established  for  the  program  was  the 
input  and  output  G’s  for  the  test  program.  These  values  of9  ,  Gp  and  0^  (note, 
in  this  instance  =  Gr,  see  Figure  3)  were  already  obtained  by  comparing 
the  well-documented  reliability  test  results  in  the  various  phases  of  the 
reliability  test  program.  Those  costs  associated  with  the  corrective  actions 
resulting  from  the  test  program  that  required  reassessment  of  the  design  and 
parts  program  work  elements  are  included  in  the  overall  test  cost. 

Gf,  the  output  MTBF  of  the  test  program,  and  the  final  or 
ultimate  MTBF  achieved  through  Program  E's  RDT&E  reliability  program  is 
that  MTBF  measured  in  the  Reliability  Demonstration  (Qualification)  Test 
Phase  of  the  reliability  test  program.  For  Program  E,  this  MTBF  was  de¬ 
termined  as: 


0 


-  e  ^  =  141  hours 


(3)  MTBF  Contribution  of  a  Reliability  Parts  Program 


Upon  review  of  the  test  failures  and  test  times  associated  with 
the  environmental  screening  and  reliability  evaluation  test  (RET)  phase  of 
the  test  program,  the  input  MTBF  of  the  test  program,  0  p  (9p  is  the  output 
of  the  parts  program,  see  Figure  3)  was  determined.  This  was  accomplished 
by  using  the  test  times  of  the  equipment  screening  and  RET  phases  and  all 
failures  accumulated  in  this  time  period.  This  effectively  included  inherent 
failures  from  isolated  sources  removed  through  equipment  screening  and 
failures  of  a  pattern  or  correctable  nature  which  were  removed  through  the 
RET  effort.  For  Program  E,  this  MTBF  was  determined  as: 

0  input  to  test  —  o  =4  hours 

P 
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and  the  incremental  gain  due  to  the  reliability  test  effort  is: 

141 

G,  =  e  /e  =  =  35 

t  t/  p  4 

(4)  MTBF  Contribution  of  a  Reliability  Design  Program 

The  next  determination  made  waB  0p ,  the  MTBF  at  the  be¬ 
ginning  of  the  parts  program  (the  output  of  the  design  program) .  This  de¬ 
termination  was  accomplished  by  evaluating  the  failure  rate  reduction 
achieved  (MTBF  improvement)  through  the  parts  program . 

This  effort  involved  a  comparison  of  part  failure  rates.  The 
part  failure  rates  experienced  in  Program  E  were  compared  with  the  failure 
rates  of  similar  items  procured  with  no  particular  attention  given  to  the 
achievement  of  high  reliability. 

A  primary  consideration  of  the  analysis  was  a  failure  rate 
comparison  of  nonstandard  parts  procured  to  Program  E  imposed  part  dis¬ 
ciplines  to  those  which  are  procured  to  minimal  military  quality  and  func¬ 
tional  standards.  For  instance,  the  Impact  on  overall  equipment,  reliability 
of  imposing  JAN-TX  and  ER  level  screening  testing  to  small  nonstandard 
electrical  components  and  stringent  environmental  screening  and  qualifi¬ 
cation  tests  on  specialty  and  major  procurement  items  on  Program  E  was 
assessed.  The  result  of  this  assessment  was  that  the  MTBF  improvement 
(0  imp)  achieved  through  the  Program  E  parts  program  was  about  2 , 5  hours. 
Therefore,  0  the  output  MTBF  of  reliability  design  program  was  deter¬ 
mined  from  the  expression: 

0  -0.  =  4.0  -  2.5=  1.5  hours 

p  imp 

(5)  Initial  Equipment  MTBF 

The  initial  reliability  assessment  of  the  Program  E  equipment 
showed  that  the  proposed  functional  deRign  would  result  in  an  average  part 
failure  rate  of  about  0.75  failure  per  10®  hours.  This  failure  rate  was  un¬ 
acceptably  high  in  view  of  the  contract  MTBF  requirement  of  the  equipment. 
Through  an  integrated  reliability  design  effort,  which  included  FMEA,  and 
audit  of  proposed  design  construction  specifications,  design  adjustments 
were  implemented  and  design  application  part  stress  levels  criteria  were 
appropriately  reduced.  Critical  and/or  potentially  high  failure  rate  items 
were  identified  and  their  impact  on  the  equipment  reliability,  along  with  their 
maximum  tolerable  failure  rates  were  identified  to  the  parts  program  re¬ 
liability  engineers  for  their  guidance  and  follow-up  action. 
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A  final  reliability  assessment  was  performed  after  the  equip¬ 
ment  design  had  been  reconfigured  to  conform  to  the  constraints  developed 
by  the  reliability  design  effort.  This  assessment  showed  a  predicted  average 
part  failure  rate  of  about  0.50  failure  per  10®  hours.  This  reduction  in 
average  part  failure  rate  (from  0.75  to  0.50  f/106  hours)  results  in  an 
MTBF  improvement  (9|mp)  of  0.5  hour.  From  the  relationship: 

e.  »  e  .  -  e, 

i  d  imp 

and  the  value  of  6  ^  is  therefore: 

-  1.5-0. 5  =  1.0  hour 

In  summary,  using  the  procedures  described  above,  the  in¬ 
cremental  reliability  (MTBF)  of  each  reliability  element  and  the  quantized 
gain  of  each  element  for  Program  E  are  as  follows: 

0  -  1.0  hour 

0^  =  1.5  hours 

0  =4.0  hours 

P 


=  141.0  hours 


The  incremental  MTBF's  and  gains  of  the  other  programs 
were  established  in  a  similar  manner.  The  results  of  these  analyses  for  ail 
10  equipments  are  summarized  in  Table  IV. 

It  should  be  noted  that  there  is  continuous  feedback  between 
the  various  major  reliability  elements  (design ,  parts  and  test).  For  example, 
the  results  from  a  comprehensive  reliability  evaluation  testing  program  pro¬ 
vide  significant  data  to  allow  for  further  improvement  in  the  equipment 
design  and  the  component  parts  utilized  in  the  design. 
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TABLE  IV.  INCREMENTAL  RELIABILITY  MTBF  AND  GAIN  DATA 


Resultant 

Equipment 

MTBF 

* 

1350.0 

225.0 

188.0 

225.0 

141.0 

501.0 

46.0 

287.0 

133.0 

209.0 

Test 

Gain 

G  -  — 
Gf  9 

P 

14.2 

20.5 

11.8 

8.7 

35.3 

7.5 

2.3 

19.1 

5.3 

3.5 

43  c  sPL'53 

So  "p. 

O 

4.8 

3.7 

3.2 

3.3 

2.7 

4.5 

2.5 

3.0 

3.6 

2.7 

ft  TJ 1  »H 

op  c  ®  CD 

■„ 

o 

H 

Design  +  Parts 
+  Test 
MTBF 
* 

6t 

1350.0 

225.0 

188.0 

225.0 

141.0 

501.0 

46.0 

287.0 

133.0 

209.0 

Design  + 
Parts 
MTBF 

0P 

95.0 

11.0 

16.0 

26.0 

4.0 

67.0 

20.0 

15.0 

25.0 

60.0 

Initial  + 

Design 

MTBF 

a 

20.0 

3.0 

5.0 

8.0 

1.5 

15.0 

8.0 

5.0 

7.0 

22.0 

Off-the 

Board 

MTBF 

ei 

2.0 

3.0 

5.0 

1.0 

11.0 

5.0 

2.0 

3.5 

15.0 

Equipment 

Letter 

Code 

A 

B 

C 

D 

E 

F 

G 

H 

J 

K 

18 


60%  LCL 


1963  1964  '  1965  '  1966  '  1967  '  1968  '  1969  '  1970  '  1971  '  1972 

DEVELOPMENT  TIME  SPAN  (YEARS) 


Figure  1.  Equipment  Code  vs  Development  Time  Span 
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SECTION  m 


ANALYSES 


1.  ANALYTICAL  APPROACH 

Two  basic  approaches  employing  the  same  analytical  techniques  have 
been  used  in  this  study  to  quantify  reliability  as  a  function  of  cost.  Both  of 
these  approaches  involve  multiple  regression  analysis  using  the  least 
squares  fitting  technique. 

The  first  approach  was  to  relate  resultant  equipment  reliability 
(MTBF)  to  the  reliability  element  costs  (incorporating  normalization  factors 
as  discussed  previously).  This  analysis  effort  utilized  resultant  measured 
equipment  reliability  (MTBF)  converted  to  a  60  percent  lower  confidence 
level  (LCL)  and  cost  data  by  element.  The  cost  data  was  collected  in  terms 
of  dollars  and  converted  to  mandays  using  average  AESD  manday  rates  ap¬ 
plicable  during  the  development  time  period  for  the  particular  equipment. 
Figure  2  of  Section  II  shows  the  development  time  spans  for  the  equipments 
in  the  study.  The  reliability  manpower  associated  with  each  program  is 
shown  in  Table  III. 

The  second  approach  was  to  relate  the  reliability  contribution  of  each 
reliability  element  to  the  cost  of  performing  that  reliability  element  (this  is 
the  same  cost  base  described  above) .  As  indicated  in  Section  II,  historical 
failure  rate  data  and  Engineering'  assessment  were  used  to  determine  the 
improvement  in  equipment  MTBF  as  a  result  of  the  contribution  of  each  of 
the  reliability  elements.  This  analysis  alBo  allowed  the  determination  of 
the  MTBF  (0  j)  that  the  equipment  would  have  achieved  without  the  benefit  of 
any  reliability  effort. 

2.  ANALYTICAL  MODELS 

It  was  assumed  that  the  relationships  between  cost  and  reliability  could 
be  explained  by  linear  equations  of  the  following  form: 

°Ri  =  Po  +  ^1  +  82  C2l  +  33  C3i  +  ‘  "  +  Pp  Cpi 

+  Vl  Xli  +  '  "  +  %+kXki  +  ®i  (1) 
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where: 


where: 


where: 


Ri 


'Pi 


P’ 


=  resultant  equipment  MTBF  (hours) 

=  reliability  element  cost  (mandays) 

=  other  prediction  parameters 
0  .  =  regression  coefficients  to  be  determined  / 

P+K  j 

=  random  variation  of  observations  (normally  distributed 
with  constant  standard  deviation  for  all  combinations  of 
Independent  variable  values) 

=  P1  eRl  +  @2  X11  + —  +  Pl+k  *111  +  °1  (2) 
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ei  * 


0. 


Ei 


reliability  element  cost  (mandays) 
resultant  equipment  MTBF  (hours) 
other  prediction  parameters 
*  regression  coefficients  to  be  determined 
random  error  (normally  distributed) 


a  + 


01Ci  + 


02  xu  +  *..+01+k  x,d  + 


(3) 


0Ei  =  element:al  reliability  worth  (MTBF  in  hours) 
Cj  =  reliability  element  cost  (mandays) 

=  other  prediction  variables 
a.'s>  0's  =  regression  coefficients  to  be  determined 
=  random  error  (normally  distributed) 
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The  form  of  these  models  is  linear  with  respect  to  the  regression  co¬ 
efficients  but  not  necessarily  linear  with  respect  to  the  independent 
variables.  This  linear  condition  allows  the  use  of  a  least  squares  procedure 
for  estimating  the  regression  coefficients. 

A  multiple  regression  program  called  MUL-REGRESSION  (see 
Appendix  A)  within  the  GE -developed  statistical  time-Bhare  program 
STATSYST***  was  used  to  estimate  the  coefficients  and  obtain  goodness  of  fit 
indications  such  as  student-t  statistics,  F-ratios,  standard  deviations, 
standard  error  of  estimates  and  various  correlation  statistics .  These  in¬ 
dicators  were  used  to  perform  significance  tests  in  order  to  assure  that 
enough  Independent  variables  were  included  to  account  for  a  significant  amount 
of  the  error.  Also,  they  were  used  to  justify  excluding  the  prediction 
variables  that  did  not  make  a  significant  contribution  to  error  elimination. 


SECT  ION  IV 


RESULTS  OF  ANALYSES 


1.  GENERAL 

It  must  be  pointed  out  that  the  following  results  are  based  on  analyses 
of  data  collected  on  ten  equipments  from  two  manufacturers.  Consequently, 
because  of  the  limited  nature  of  the  data  base,  caution  must  be  exercised  in 
utilization  of  the  results.  Specifically,  caution  must  be  taken  to  apply 
equations  presented  within  the  bounds  of  the  data  from  which  the  equations 
were  generated.  Section  n  summarizes  the  data  base  for  the  study  and 
should  be  used  as  a  guide  for  the  range  of  use  of  the  equations. 

The  results  presented  in  this  section  are  divided  into  two  categories. 
The  first  category  is  termed  Gross  Prediction  Equations  and  represents 
results  derived  from  factual  equipment  characterization  and  cost  data.  The 
second  category  is  termed  Allocation  Equations  and  represents  results  de¬ 
rived  from  assessment  of  historical  failure  rate  data.  Consequently,  the 
latter  equations  are  less  rigorous  than  the  former. 

2  .  GROSS  PREDICTION  EQUATIONS 

2 

Multiple  regression  analyses  were  performed  to  develop  prediction 
equations  for: 

•  Total  reliability  cost  as  a  function  of  resultant  equipment  MTBF 
and  quantity  of  parts . 

3 

•  Reliability  prediction  cost  as  a  function  of  quantity  of  parts. 

3 

•  Reliability  design  review  cost  as  a  function  of  quantity  of  parts . 

•  Reliability  failure  modes  and  effects  analysis  cost  as  a  function  of 
the  prediction  cost. 

•  Reliability  design  cost  as  a  function  of  quantity  of  parts. 


— 

See  Appendix  A. 

9 

Note  that  prediction  and  design  reviews  are  mandatory  per  MIL-STD-785 
and  design  review  and  prediction  are  directly  relatable  to  equipment 
complexity. 
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•  Resultant  equipment  MTBF  as  a  function  of  reliability  parts  pro¬ 
gram  cost,  reliability  test  program  cost  and  quantity  of  parts. 

All  of  the  cost  data  used  in  developing  these  relationships  was  normalized  as 
described  in  Section  IT  prior  to  performing  the  analyses. 

It  should  also  be  pointed  out  that  a  formal  failure  modes  and  effects 
analysis  effort  was  performed  for  only  throe  of  the  equipments  in  the  data 
base.  Therefore,  the  prediction  relationship  for  failure  modes  and  effects 
analysis  cost,  being  developed  from  only  three  pieces  of  data,  is  statistical¬ 
ly  weaker  than  the  others .  Also,  since  there  were  only  three  equipments 
with  FMEA,  these  equipments  were  excluded  from  the  development  of  the 
prediction  equation  for  the  reliability  design  cost. 


The  prediction  equations  are: 

Total  Reliability  Cost 
CT  . 

Reliability  Prediction  Cost 
CpR  =  119.16  +  0.098Np 

Reliability  Design  Review  Cost 
CDR  =  92.479  +  0.022Np 

Reliability  Failure  Modes  and  Effects  Analysis  Cost 


Reliability  Design  Cost  (by  definition) 

Cd  =  CPR  +  CDR  +  CFM 

Reliability  Design  Cost  (excluding  FMEA) 

C  '  =  242.2  +  0.121  N 
d  p 


(5) 

(6) 

(7) 

(8) 


(8a) 


Reliability  Parts  Program  Cost  plus  Reliability  Test  Program  Cost 


Resultant  Equipment  MTBF 

5.36(cp1‘42Xc.°‘64') 

R  "  N  1,37  (10) 

Jr 

A  summary  of  the  key  statistics  for  these  equations  is  shown  in  Table 
V  for  equations  (4),  (5),  (6),  (7),  (8a)  and  (10).  Equations  (8)  and  (9)  are  not 
included  in  the  summary  since  they  were  generated  by  definition  instead  of  by 
performing  regression  analyses. 

These  equations  allow  prediction  of  reliability  costs  and  resultant  equip¬ 
ment  MTBF.  Assessment  of  the  contribution  of  the  reliabilitHelement  costs 
to  the  resultant  equipment  MTBF  is  also  provided.  The  contribution  of  the 
reliability  elements  to  equipment  MTBF  is  addressed  In  greater  detail  later 
in  this  report;  therefore,  these  equations  have  been  termed  gross  prediction 
equations. 


a.  Adequacy  of  Prediction  Equations 

The  statistics  provided  through  the  regression  analyses  give  an 
indication  of  the  accuracy  of  the  equations.  As  can  be  seen  from  Table  V, 
the  index  of  determination ,  R2 ,  indicates  that  the  least  accurate  prediction 
equation  accounts  for  approximately  48  percent  of  the  initial  variation.  Also, 
from  Table  V,  a  comparison  of  the  standard  error  of  estimate,  SE,  and  the 
standard  deviation,  Sy,  indicates  the  adequacy  of  the  equations. 

Also,  the  adequacy  of  these  prediction  equations  can  be  assessed 
by  comparing  the  values  predicted  by  each  equation  with  the  actual  data 
values  that  were  used  to  develop  the  relationships  (see  Table  VI).  Figures  4 
through  11  show  plots  of  the  predicted  values  vs  the  actual  values  for  equa¬ 
tions  (4)  through  (10).  The  solid  line  in  these  plots  is  the  line  of  perfect  fit. 

In  addition  to  percent  difference  between  predicted  and  actual 
values,  confidence  limits  about  the  actual  average  values  can  be  computed  by 
multiplying  the  predicted  values  by  the  appropriate  factor.  Multiplying  fac¬ 
tors  for  60,  75  and  95  percent  confidence  limits  are  shown  in  Table  VII  for 
each  of  the  allocation  equations.  These  multiplying  factors  assume  a  normal 
distribution  and  the  closer  they  are  to  1.00,  the  narrower  the  limits  and  the 
more  precise  the  estimate.  The  more  the  values  of  the  dependent  variables 
of  the  new  equipment  deviate  from  the  average  of  the  data  base,  the  wider 
the  limits  will  be. 
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TABLE  VI.  COMPARISON  BETWEEN  PREDICTED  COSTS  AND  ACTUAL  COSTS  WITH 


Note:  Equations  summarized  :n  Table  V. 


TABLE  VTL  CONFIDENCE  LIMIT  MULTIPLYING  FACTORS 


As  can  be  seen  from  Table  VI  and  Figure  5 ,  the  largest  deviation 
for  the  predicted  values  was  the  reliability  prediction  cost  for  radar  E  which 
was  high  by  a  factor  of  almost  four.  Also,  the  design  is  high  by  a  factor  of 
three.  This  results  from  the  fact  that  the  actual  reliability  prediction  and 
design  costs  for  this  equipment  are  low.  The  reason  they  are  low  is  attri¬ 
buted  to  the  fact  that  this  radar  was  developed  in  parallel  with  the  digital 
portion  of  the  same  system.  This  is  digital  equipment  F  in  the  data  base.  It 
appears  that  the  radar  benefitted  from  this  parallel  development,  which  re¬ 
sults  in  a  lower  cost  in  prediction  and  design  due  to  commonality  of  tasks. 

If  equipment  E  is  excluded  from  the  reliability  prediction  analysis,  the  index 
of  determination,  R2,  increases  from  0.488  to  0.812  and  the  standard  error 
of  estimate  decreases  from  679.29  to  430,05. 


All  of  the  other  prediction  equations  yield  predicted  values  that  are 
from  0. 5  to  2  times  the  actual  values.  It  is  felt  that  this  is  acceptable  for 
initial  estimates  based  on  the  limited  data  base  available  coupled  with  general 
equipment  development  uncertainty. 

b.  Using  the  Gross  Prediction  Equations 


The  gross  prediction  equations  have  two  purposes: 

(1)  To  predict  the  total  reliability  cost,  C-p,  required  to 
achieve  a  given  MTBF  requirement, 

(2)  To  predict  the  achievable  reliability,  by  the  ex¬ 
penditure  of  certain  levels  of  resources  on  given 
reliability  program  elements. 

For  analog  equipments ,  the  first  purpose  can  be  realized  through 
the  direct  use  of  equation  (4).  In  addition,  equations  (5),  (6),  (7),  and  (8a) 
can  be  utilized  to  break  out  design  cost.  In  order  to  approximately  isolate 
parts  and  test  costs,  equation  (10)  can  be  used  with  6^  equal  to  the  require¬ 
ment  MTBF  as  described  later. 


For  digital  equipments,  define  using  equations  (4)  through  (9) 
as  required.  Then,  using  equation  (10)  with  Gr  equal  to  the  requirement 
MTBF,  determine  the  values  of  Cp  and  Cq.  The  value  of  Cp  divided  by  K 
yields  the  digital  parts  cost.  Next,  divide  the  value  obtained  for  Cj  (or  Cj' 
as  applicable)  by  K.  The  total  cost  C.p  for  digital  equipments  is  then  equal  to 


+  C 


t 
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To  achieve  the  second  purpose,  where  a  given  dollar  figure  is 
allotted  to  the  reliability  effort  and  it  is  desired  to  determine  the  reliability 
achievable  and  the  necessary  resource  allocations  to  be  made  to  each  of  the 
reliability  elements,  use  the  following  procedure.  Equations  (5),  (6),  (7), 

(8),  (8a)  are  used  to  define  the  fixed  design  costs.  Then  equation  (10)  is 
utilized  with  various  combinations  of  Cp  and  C*  to  determine  the  resulting 
values  of  reliability  possible.  Again,  modifying  factors  for  digital  equip¬ 
ments  must  be  used  as  required  on  the  various  cost  parameters. 

A  specific  example  of  this  application  of  equation  (10)  is  portrayed 
in  Figure  12.  In  this  example,  the  resultant  equipment  MTBF  requirement 
is  chosen  to  be  188  hours,  the  quantity  of  parts,  Np,  is  chosen  to  be  11,160, 
the  total  reliability  program  cost  is  predicted  from  equation  (4)  to  be  7,385 
mandays  (MD),  constant  for  this  example,  and  the  reliability  design  program 
cost  is  predicted  from  equations  (5),  (6),  (7)  and  (8)  to  be  1,518  MD's.  From 
equation  (9),  this  leaves  5,867  MD's  to  be  divided  between  the  reliability 
parts  program  and  the  reliability  test  program.  The  curve  of  Figure  12  is 
obtained  by  varying  the  parts  cost  to  test  cost  ratio,  while  holding  the  total 
cost  for  these  two  elements  constant  at  5,867  MD's.  The  solid  portion  of  the 
curve  shows  the  range  of  the  data  base  of  the  study.  The  asterisks  show  the 
60 -percent  confidence  limits^  about  the  data  points. 

As  can  be  seen  from  the  gross  prediction  equations,  one  would 
expect  approximately  1.5  times  as  much  to  be  spent  in  parts  as  in  test  to 
obtain  the  maximum  MTBF.  The  data  base  for  the  10  equipments  showB  that 
1.2  times  as  much,  on  the  average,  was  spent  in  parts  as  in  test.  This 
represents  the  initial  parts  cost.  The  remainder  of  the  parts  cost,  which 
results  from  test  feedback  (see  Figure  3)  iB  included  in  test  cost.  This  feed¬ 
back  parts  cost  is  made  more  obvious  in  the  next  section  through  incremental 
gain  assessment,  where  it  is  shown  that,  as  test  cost  increases,  the  ratio  of 
parts  cost  to  test  cost  decreases  and  the  MTBF  increases.  Therefore,  in 
actual  practice,  more  will  be  spent  in  test  than  in  parts. 

Furthermore,  there  is  a  range  of  MTBF's  that  can  be  achieved  for 
a  specified  equipment  depending  on  the  amount  of  dollars  expended  and  the 
design  constraints  that  are  within  the  state  of  the  art.  Therefore,  equation 
(4)  can  be  used  to  develop  a  relationship  to  assess  the  impact  on  the  increase 
in  cost  as  a  function  of  the  Increase  in  MTBF  expressed  as  a  ratio  of  the 
desired  MTBF  vs  minimum  MTBF,  Using  equation  (4) 


4 

A  confidence  band  (or  limits)  on  the  mean  of  the  dependent  variable  for  a 
specified  set  of  values  of  the  independent  variables  (data  points)  is  an 
interval  which  one  can  state  with  a  specified  degree  of  confidence  contains 
the  true  mean  (or  long  range  average)  value  of  the  dependent  variable 
(Ref.  3). 
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where 


1.804 

1.804 


0.684 

07684 


'l 


/ 

C_  =  cost  (normalized  and  expressed  in  mandays 
required  to  achieve 

C_b  =  cost  (normalized  and  expressed  in  mandays 
required  to  achieve  ^ 

e„  =  desired  MTBF  within  the  state  of  the  art 
a 

@  b  =  minimum  MTBF  within  the  state  of  the  art 

If  N  is  constant 
P 


This  relationship  has  been  used  to  produce  the  curve  shown  in 
Figure  12a.  This  curve  can  be  used  to  obtain  increase  in  cost  as  a  result  of 
an  increase  in  MTBF. 

3.  ALLOCATION  EQUATIONS 

This  paragraph  details  the  effort  set  forth  to  quantize  the  reliability 
contribution  of  each  reliability  element  in  terms  of  MTBF  hours  as  function 
of  cost. 

The  first  step  toward  establishing  relationships  between  reliability  and 
cost  using  the  data  from  Section  II  v/as  to  develop  a  usable  starting  point 
relationship  involving  initial  MTBF  (see  Figure  13).  This  initial  MTBF  is 
the  reliability  of  the  equipment  that  would  be  realized  without  any  reliability 
effort  being  applied.  A  multiple  regression  was  performed  and  a  prediction 
equation  for  was  developed  in  terms  of  the  equivalent  analog  quantity  of 
electrical  parts  (Npjc),  The  normalization  factor  used  here  was  three  since 
the  average  part  failure  rate  for  the  analog  equipments  was  approximately 
three  times  the  average  part  failure  rate  for  the  digital  equipments  of  the  10 
equipments  in  the  data  base.  The  equation  is: 

ei  =  1.061  x  104/NpK0, 921 
and  is  plotted  in  Figure  13. 
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Next,  effective  cumulative  reliability  MTBF  relationships  were  de¬ 
veloped  by  performing  regression  analyses  using  the  cumulative  reliability 
element  MTBF's  as  the  dependent  variable  and  the  individual  normalized 
reliability  element  costs  along  with  the  previous  cumulative  reliability 
element  MTBF's  as  the  independent  variables  (see  Figure  14).  The  data  for 
the  10  equipments  is  summarized  in  Tables  III  and  IV. 

MTBF  gain  relationships  were  also  developed  using  the  same  data. 

The  gains,  MTBF  out  divided  by  MTBF  in  (see  Figure  14),  were  used  as  the 
dependent  variables  and  the  normalized  reliability  element  costs  were  used 
as  Independent  variables.  These  MTBF  and  gain  equations  along  with  the  key 
statistics  are  summarized  in  Table  Vm.  Plots  of  the  gain  equations  are 
shown  in  Figure  15. 

a.  Adequacy  of  the  Allocation  Equations 

The  statistics  provided  through  the  regression  analyses  give  an 
indication  of  the  accuracy  of  the  equations.  As  can  be  seen  from  Table  VIII 
the  index  of  determination,  R^,  indicates  that  the  prediction  equations 
account  for  a  significant  amount  of  the  initial  variation.  Also  from  Table 
VIII,  a  comparison  of  the  standard  error  of  estimates,  Sj;,  and  the  standard 
deviations,  Sy,  indicates  the  adequacy  of  the  equations. 

The  adequacy  of  these  equations  can  be  assessed  by  comparing  the 
values  predicted  by  ihe  equations  with  the  actual  data  values  for  each  equip¬ 
ment  that  was  used  to  develop  the  relationships.  This  comparison  can  be 
seen  in  Tables  IX  and  X.  Figures  16  through  23  are  plots  of  the  predicted 
values  vs  the  actual  values.  The  solid  line  in  these  plots  is  tho  line  of 
perfect  fit. 

In  addition  to  percent  difference  between  predicted  and  actual 
values,  confidence  limits  about  the  actual  average  values  can  be  computed 
by  multiplying  the  predicted  values  by  the  appropriate  factor.  Multiplying 
factors  for  GO,  75  and  95  percent  confidence  limits  are  shown  in  Table  XI  for 
each  of  the  allocation  equations.  These  multiplying  factors  assume  a  normal 
distribution;  the  closer  they  are  to  1.00  the  narrower  the  limits  and  the  more 
precise  the  estimate.  The  more  the  values  of  the  dependent  variables  of  the 
new  equipment  deviate  from  the  average  of  the  data  base  the  wider  the  limits 
will  be. 
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TABLE  Vm .  MTBF  GAIN  EQUATIONS  WITH  STATISTICS  SUMMARY 


TABLE  X.  COMPARISON  BETWEEN  PREDICTED 
AND  ACTUAL  GAINS  WITH  PERCENT  DIFFERENCE 


TABLE  XI.  CONFIDENCE  LIMIT  MULTIPLYING  FACTORS 


Note:  Equations  summarized  in  Table  vm. 


b.  Using  the  Allocation  Equations 

In  summary,  the  allocation  equations  developed  in  this  study  are 
as  follows: 


ei  * 

(1.061  x  104)/Npk0'921 

(il) 

ed  ■ 

Mu(»1°'#"')  (cb0-300) 

(12) 

ed  - 

0.58S(9d1-13i)(cp°'185) 

(13) 

9t  ■ 

0.094  ( e°-m' )  (c,0’741') 

(14) 

II 

o° 

0.302(Cd0,247S) 

(15) 

II 

l-US^0,13*) 

(16) 

Gt  * 

0.0064(Ct0,952S) 

(17) 

gt  - 

(Gd)  (Gp)  (Gt) 

(18) 
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Equation  (11)  can  be  used  to  estimate  the  initial  MTBF  prior  to 
performing  any  reliability  effort.  To  use  equation  (11),  the  number  of  elec¬ 
trical  parts  must  be  normalized  to  an  analog  level.  This  is  done  for  digital 
equipments  by  dividing  the  estimate  of  the  number  of  electrical  components 
by  three. 


Equations  (12)  through  (14)  can  be  used  to  predict  the  cumulative 
MTBF  contribution  due  to  the  various  reliability  elements.  Specifically, 
since  these  equations  were  developed  independently  of  each  other  using  the 
data  base  of  Section  n,  the  output  of  equation  (11)  can  be  used  in  equation  (12) 
to  predict  the  cumulative  MTBF  after  the  reliability  design  effort;  the  output 
from  equation  (12)  can  be  used  in  equation  (13)  to  predict  the  cumulative 
MTBF  after  the  reliability  design  plus  parts  effort;  and  the  output  from  equa¬ 
tion  (13)  can  be  used  in  equation  (14)  to  predict  the  cumulative  MTBF  after 
the  reliability  design  plus  parts  plus  test  effort.  This  output  of  equation  (14) 
is  also  the  predicted  resultant  MTBF  for  the  equipment.  This  approach 
yields  a  predicted  MTBF  that  is  dependent  on  the  total  reliability  cost  chosen 
and  how  it  is  spread  among  the  reliability  elements.  On  the  average,  for  the 
10  equipments  in  the  study,  22  percent  of  the  cost  was  put  into  design,  40 
percent  into  parts  and  38  percent  into  test.  A  little  later  in  this  section,  the 
gain  relationships  will  be  used  to  assess  the  effect  on  equipment  reliability 
as  a  result  of  varying  these  percentages. 

If  one  is  interested  in  a  prediction  of  the  MTBF  contribution  of 
each  of  the  individual  reliability  elements,  these  can  be  obtained  by  subtract¬ 
ing  @1  from  to  get  the  design  contribution;  by  subtracting  ^  from  0p  to 
get  the  parts  contribution;  and  by  subtracting  from  &  to  get  the  test  con¬ 
tribution. 


Equations  (15)  through  (17)  can  be  used  to  predict  the  reliability 
gain,  i.e. ,  the  times  improvement  in  MTBF  as  a  function  of  the  reliability 
element  costs.  Specifically,  selected  costs  for  each  reliability  element  can 
be  used  in  equations  (15),  (16)  and  (17)  to  predict  the  element  MTBF  gains. 
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Further,  as  indicated  in  equation  (18),  hese  gain  equations  can  be  multiplied 
together  to  yield  an  expression  for  tota1  gain  in  terms  of  the  reliability 
element  cost-  When  this  is  done,  equation  (18)  can  be  written  as: 


0.002l(cd 


0.247 


(19). 


Multiplying  and  dividing  each  term  on  the  right  side  by  CT,  *he  total  cost, 
it  follows  that : 


[>»•»« 

c  .0.137 

0.952  1.336 

/  Li  v  /  \ 

Gt  =  0.0021 

1 

i 

H 

7\ r 

/ 

(20) 


Thla  aquation  now  can  ba  usad  to  assess  tha  Impact  of  changing  tha 
distribution  of  total  effort  within  th*  reliability  elements.  This  la  done 
by  determining  a  total  reliability  coat.  For  example,  it  daalred  ae  a  flrat 
cut,  aquation  (4)  can  be  ueed  for  thle  purpose.  Thle  coat  le  than  divided 
among  the  reliability  element!  in  keeping  with  tha  average  of  the  10  equip** 
manta  In  the  study.  By  holding  tha  total  coat  in  tha  aquation  constant  while 
varying  tha  ratloe  of  tha  alamsnt-cost-to-total  cost  over  the  date  range,  the 
effect  on  total  gain  can  ba  observed.  In  doing  thle,  the  eum  of  reliability 
element  cost  ratios  la  tho  equation  must  elwaya  equal  one.  For  example,  if 
50  percent  of  the  coat  le  put  in  teat  and  30  parcant  in  parta,  than  20  par-r 
cant  muat  be  put  In  design.  A  display  of  such  an  assessment  of  varying 
reliability  element  costs  is  shown  in  Figure  24  for  e  total  cost  of  7680  man- 
daya  which  rapreaanta  the  average  of  Che  total  coat  for  the  10  equipments  in 
the  study.  The  rectangle  in  the  figure  repreeenta  the  range  of  the  data. 

Tha  vertical  and  tha  horlsontal  axes  reprsssnt  the  percent  of  total  cost  In 
parts  and  test  respectively.  The  diagonal  lines  represent  the  percentage  of 
total  cost  In  deeign.  The  contour  lines  represent  conatant  total  gain  lines 
for  varying  percentage  distributions  of  the  three  reliability  elements.  As 
can  ba  aaen  from  Figure  24,  test  la  the  most  effective  single  element.  Also 
the  moat  optimum  gain  within  the  range  of  the  data  le  obtained  when  twice  as 
much  effort  le  expended  in  test  as  in  parte.  Perhaps  this  ratio  should  be 
even  higher  but  the  limited  range  of  data  in  this  study  does  not  allow 
speculation  beyond  tha  2  to  1  ratio. 
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LABILITY  COST 


ACTUAL  COST  (MANDAYS) 

Predicted  R  Prediction  Cost  vs  Actual  R  Prediction  Cost 
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Figure  12.  Resultant  Equipment  MTBF  vs  Ratio  of  Parta  to  Test  Cost 
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Figure  12a.  Cost  Ratio  vs  MTBF  Ratio 
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MTBF  vs  Equipment  Complexity 


CUMULATIVE  EFFECTIVE  MTBFS 
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Incremental  Reliability  Model 


Predicted  Off -the -Board  MTBF  vs  Actual  Off-the- Board  MTBF 


Predicted  Cumulative  R  Design  MTBF  vs  Aetna!  Cumulative  R  Design  MTBF 


ACTUAL  MTBF 

e  19.  Predicted  Cumulative  R  Test  MTBF  vs  Actual  Cumulative  R  Test  MTBF 
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'e  23.  Predicted  Total  R  Gain  vs  Actual  Total  R  Gain 


SECTION  V 


CONCLUSIONS  AND  RECOMMENDATIONS 


Significant  relationships  have  been  developed  lor  predicting  incre¬ 
mental  reliability  costs,  MTBF's  and  gains  in  terms  of  the  contractual 
resultant  equipment  MTBF  requirement  and  equipment  complexity. 

The  validity  of  the  relationships  as  applied  to  a  future  equipment  will 
depend  on  the  performance  requirements,  design  criteria,  and  design  disci¬ 
plines  of  the  new  equipment  being  within  the  bounds  of  the  data  base  and  con¬ 
straints  as  defined  in  this  report. 

The  best  measure  of  accuracy  of  the  prediction  relationships  is 
through  consideration  of  statistical  confidence  as  applied  to  the  average  of 
the  data.  Tables  VII  and  XI  of  Section  IV  show  the  percentage  ranges  of 
the  actual  average  values  within  which  the  various  predicted  values  will  fall 
for  a  typical  airborne/space  equipment  with  60,  75  and  95  percent  confidence. 
The  conservative  75  percent  confidence  interval  is  recommended  since  the 
actual  data  was  limited  to  10  equipments. 

The  prediction  parameters  chosen  here  seem  to  account  for  a  signi¬ 
ficant  proportion  of  the  error  when  predicting  incremental  reliability  costs 
and  MTBF's.  Although  no  proof  or  disproof  of  other  significant  variables 
was  found  based  on  the  data  of  this  study,  the  user  of  the  equations  must  be 
aware  of  the  possibility  of  such  variables  and  consequently  apply  sound 
engineering  judgment  and  caution  throughout  their  use. 

The  relationships  developed  in  this  study  can  be  used  to  predict  in¬ 
cremental  reliability  coats  with  associated  incremental  gain  as  well  as  to 
make  engineering  trade-off  decisions  with  respect  to  a  sound  reliability 
program. 

The  efforts  of  this  study  have  shown  that  cost  and  reliability  are  related. 
It  is  strongly  recommended  that  the  data  base  be  expanded  to  strengthen  and 
broaden  the  utility  of  the  relationships.  The  broadening  of  the  data  base 
should,  if  possible,  include  not  only  more  equipments  of  the  same  type,  but 
also  equipment  types  other  than  airborne/space  and  provide  for  more  data 
to  allow  a  more  detailed  analysis  of  modified  and  digital  designs. 
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APPENDIX  A 

MUL- REGRESSION  COMPUTER  PRINTOUT 


This  appendix  contains  a  typical  computer  printout  of  the  multiple 
regression  program,  MUL- REGRESSION,  that  was  used  for  the  bulk  of  the 
regression  analyses  in  the  study.  This  program  is  part  of  the  time-share 
statistical  program  called  STATSYST***  developed  by  the  General  Electric 

Company. 

RUN  MUL-REGRESS I  ON  <LKRVALC>LMTBFjLQUAPTSB> 


VERS10N  16  AUG  71  MULTIPLE  REGRESSION 
INTERCEPT  7YES 


INDEPENDENT 

VARIABLE 

lmtbf 

lquaptsb 

intercept 


coefficient 

3.701 6232  IE-01 
6.8 4466376E-01 

S.B9625701E-0I 


CHECK 

3.31E-09 

-7.S0E-08 


STANDARD 

ERROR 

7 . 42941 E-02 
8  > 1 2490E-02 


T-STAT1STIC 

4.98239Ei-00 

8.4243oe+oo 


CODE  ?1 


CASK 

OBSERVED 

PREDICTED 

NO. 

VALUE 

VALUE 

RESIDUAL 

7.  DEVIATION 

1 

1 .00626E  +  01 

9 . 98  369E+00 

-7.B9061E-02 

-0.  79 

:  2 

9 • 38824E+00 

9 . 36S35E>00 

-2.2881 4E-02 

-0.24 

i 

6*821 1 4E*00 

9. 1  I238E+00 

2.9U35E-01 

3.20 

i  J 

A 

9 . 23640E*00 

8  <  94S20E+00 

-2.9 1202E-01 

-3.26 

\  5 

8 . 67989E*Q0 

8 .86B96E+00 

1 .09O7IE-O1 

2.13 

i 

i  6 

8 .951 44E*00 

8.90725E+00 

-4.4I91BE-02 

-0.50 

V 

7 • 998  34E+00 

7.9S498E*00 

- 1 . 33590E-02 

-0.  17 

*.j  R 

7.93559E*00 

7  .94745E*00 

1 . 18656E-02 

0.15 

9 

7.83676E*00 

7.891  70E  +  00 

S.49358E-02 

0.70 

10 

7 .880.3E+00 

7  •  78 36 AE+OO 

-9 . 67677E-02 

-1.24 

SUM  0F  RESIDUALS  ■ 
STANDARD  ERROR  OF  THE  ESTIMATE  « 


-6. S5651E-07 


1  . 8  0033E-0 I 
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CflUE  7  2 


ANALYSIS  OF  VARIANCE 


m  '  ni 

SOURCE 

OF 

SS 

MS 

RELHESSlON 

error 

2 

'  7 

4.9  S8&2E+00 
2.26B84E-0I 

2.47931E+00 

3.24120E-02 

101AL 

9 

5.  18551E+00 

F- RATIO  • 

7 . 6 A936E+0 1 t 

A  100.00  *  VALUE 

MULIlHLk,  hb.LKti>SI  ON  COEFF.  »  0.977878556 

CO UK  7 A 


CCVAkl  A'ICb.  MATRIX 


7 .737E-01 
2.F00E-01 
4.7R0E-01 


2.8O0E-OI 

6.469E-01 

5.464E-01 


4.780E-01 
b. A6AE-0I 
5.762E-01 


<LMTPF  7 
{LflUAP ISH1 
(Ltd'vAl.C  1 


V»h  I  AHI.F 


(Lk I HK  ) 

CL0UAH1SH) 
(LRHVALC  5 


CODE  7S 


MEAN 


5.41 627EtOO 
R  .llrtQ«9fc>00 
M . 6790AE+00 


STANDARD 
OKVl AT  I  ON 


R.795B4E-01 
H.04291E-01 
7  .S90S7E-0! 


CORRELAT I  ON  MATRIX 


<LMTBF  > 
(LQUAPtSO) 
(LKKVALC  ) 


1 .0000000 
0.3958072 
0.7  1  5999B 


0.3958072 
1 .0000000 
0.8950321 


0.7159998 
0.8950321 
1 .0000000 
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CONE  IDENCE  BANDS  AND  PHEQI  CT  1  ON  INTERVALS  FOR  E(Y) 

it  DATA  VALUE  <  S>  »  N3.  Or  CONE  1  DENCE  LEVELS*  t  LEVELS  --  ?5 . 41 627 » B . 839 49 
tj, .9b» • 7  S» .60 

UU.mc  IDENCE  BANCS  7YES 


CONFIDENCE  SANDS  FOR  E(Y7 


CONFIDENCE 

LOWER 

PREDICTED 

UPPER 

BAND 

LEVEL 

POUND 

VALUE 

BOUND 

WIDTH 

9S.C0 

K . S4446E*00 

8 . 67908E+00 

8  « 8  1  370E+00 

2.69244E-01 

VS.  00 

8  »60767E*00 

B. 67908  K«  00 

8.7  S049E+00 

1 . 4231 6E-01 

60. 00 

8 .62307E+00 

8 -67908C+00 

3  •  7 30Q9E+00 

I . 02026E-01 

PREDICTION  INTERVALS  YNO 

CODE  70 

stork  predicted  values  in  statsysteh  ?nb 


i.EADY 
Iti  I  CP 
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APPENDIX  B 


RELIABILITY  PLANNING  AND  MANAGEMENT  (RPM)  METHODOLOGY 


This  appendix,  a  reprint  of  Reference  5,  presents  a  new  approach  to 
the  reliability  planning  and  management  of  complex  weapon  systems.  This 
methodology  was  developed  and  is  being  used  by  the  Aerospace  Electronic 
Systems  Department  of  the  General  Electric  Company. 
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RELIABILITY  PLANNING  AND  MANAGEMENT  -  RPM 


J.D.  Selby,  General  Manager 
and 

S.G.  Miller,  Manaeer-ReUabilUy  and  Quality  Assurance 

General  Electric  Company 
Aeroapace  Electronic  Syatema 
Utica,  New  York 


INTRODUCTION 

This  paper  presents  a  new  approach  to  the 
reliability  planning  and  management  of  complex 
weapon  systems.  To  set  the  stage,  It  is  neces¬ 
sary  that  certain  basic  assumptions  be  accepted. 
First,  It  must  be  assumed  that  lor  any  new  weap¬ 
on  system  a  realistic  reliability  model  has  been 
developed.  Secondly,  it  must  be  assumed  that  the 
modol  reflects  a  reasonable  apportionment  of  the 
reliability  mean  time  between  failure  (MTBFJ  re¬ 
quirements  among  the  various  subsystems  or  e- 
qulpmunts.  In  other  words,  we  assume  that  when 
a  request  for  proposal  (RFP)  is  released  for  bid, 

It  contains  a  reliability  requirement  that  ran  bo 
met. 

STATEMENT  OF  THE  PROBLEM 

The  problem  I  will  address  is  the  credibility 
In  avionics  reliability  planning  and  acquisition 
management.  Experience  has  demonstrated  that 
to  a  varying  degree,  a  reliability  program  credi¬ 
bility  gap  exists  between  stated  equipment  relia¬ 
bility  requirements  and  realized  or  realizable  a- 
chlcvoment.  This  gap  stems  largely  lrom  the 
lack  of  u  uniform  method  of  reliability  program 
structuring  and  evaluation.  The  result  Is  Imple¬ 
mentation  of  unrealistic  programs  with  unachiev¬ 
able  requirements.  These  shortcomings  become 
evidenced  in  a  lank  Df  understanding  between  the 
buyer  and  contractor  an  to  reliability  require¬ 
ment,  risks,  and  program  Impact  to  be  associ¬ 
ated  with  a  given  equipment  development  plan. 

Typically,  each  equipment  proposal  describes 
an  overview  reliability  program  apparently  com¬ 
pliant  with  specification  requirements.  In  a  de¬ 
tailed  cxunitnutlon  of  Individual  Implementation 
plans,  the  content  may  vary  widely,  thus  opening 
to  question  program  credibility  and  achievable  re¬ 
liability.  This  credibility  gap  Is  first  evident  In 
the  dimensioned  equipment,  resources,  facilities, 
etlorl,  time,  and  their  planned  utilization,  pro¬ 
jected  by  contractors  as  required  for  compliant 
program  execution.  Currently,  both  the  contrac¬ 
tor  proposal  manager  and  the  equipment  buyer 
lack  tools  with  which  to  evaluate  with  reasonable 
conviction  which,  if  any,  of  the  proposed  programs 
can  result  in  a  product  compliant  with  the  relia¬ 
bility  requirements. 


Evidence  as  to  the  degree  of  reliability  non¬ 
conformance  currently  being  experienced  is  shown 
by  a  review  of  the  field  performance  (Table  I)  of  a 
1968/1966  vintage  aircraft  weapons  system.  This 
systom,  romprlscd  of  new  and  modified  hardware 
procured  under  a  program  utilizing  MIL-STD-7B5, 
is  augmented  b<>  specific  requirements  for  relia¬ 
bility  qualification  testing,  reliability  product 
screening,  production  FAC  I,  and  production  reli¬ 
ability  acceptance  testing.  A  within- program  re¬ 
view  of  Individual  product  reliability  performance, 
during  the  first  quarter  of  1970,  Indicates  that  the 
equipment's  achieved  mean  time  versus  contract 
specified  was  noncompliant  by  a  factor  as  large  us 
20:1,  typically  10:1,  with  only  a  few  Bolect  prod¬ 
ucts  being  compliant.  In  addition,  reliability  per¬ 
formance  compliance  would  not  uppear  to  bo  tiod 
directly  to  a  specific  contractor  or  the  equipment 
mean  time  goal,  since  complex  and  rumple  equip¬ 
ments  are  both  compliant  as  well  as  significantly 
noncompliant.  Again,  these  results  were  exper¬ 
ienced  in  an  environment  where  the  equipments 
were  procured  to  dimensioned  reliability  require¬ 
ments  in  the  lute  60s. 

TABLE  1.  WEAPONS  SYSTEM  FIELD 
RELIABILITY  COMPARISON* 
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Having  established  that  realized  product  re¬ 
liability  Is  widely  diverse  and  typically  low,  it  is 
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necessary  to  identify  the  cause  to  facilitate  cor* 
rectiva  action  and  establish  a  method  for  evalua¬ 
ting  and  appraising  program  reliability  potential, 
prior  to  and  during  the  contract  acquisition  proc¬ 
ess,  Insight  into  the  cause  and  impact  of  the  lack 
of  reliability  achievement  may  be  gained  (Figure 

1)  from  a  review  of  the  time-phaaed  performance 
of  the  avionic  auppliers  on  a  completely  separate 
weapon  aystem  development  of  the  mid- 60s. 
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Figure  1,  A  Weapons  System  Avionic  Suppliers' 
Performance 


A  number  of  observations  become  clear  from 
this  historical  data: 

1)  The  original  weapons  system  reliability 
apportionment*  were  realistic  as  evidenced 
by  their  eventual  achievement 

2)  Timely  RAD  compliant  performance  wrb 
not  achieved 

3)  Reliability  nonconforming  production 
hardware  was  delivered 

4)  Modified  equipment*  with  their  attendant 
reduced  development  risks  achieved  com¬ 
pliance  in  a  more  orderly  and  timely  fash¬ 
ion  ttian  newly  developed  equipments. 

For  tho  newly  developed  equipments ,  Hie  initial 
average  achieved  reliability  waH  approximately 
8'V  of  their  specified  requirement.  Each  equip¬ 
ment  required  extensive  teet  evaluation  and  im¬ 
provement  prior  to  achieving  compliance.  The 
Lack  uf  compliance  was  equally  applicable  lo  a 
cross  section  of  contractor*  and  product  types 
with  complexity  variation  of  17:1.  These  results 
are  typical  ot  weapon  systems  designed  with  the 
modern  technology,  methods,  and  automation  of 
the  80s.  A  way  of  stating  the  problem  in  its  to¬ 
tality  la  that  management,  govermmint  and  indus¬ 
try,  lack  a  quantitative  yardstick  with  which  to  di¬ 
mension,  plan,  manage,  fund,  and  monitor  relia¬ 
bility  development  as  a  part  of  equipment  devel¬ 
opment. 


THE  NEED  FOR  A  SOLUTION 

The  trend  in  product  reliability  requirements 
from  the  1880s  through  the  1960s  can  be  por¬ 
trayed  (Figure  2)  by  the  reliability  plot  from  M1L- 
STD-758A  as  updatsd  and  augmentod  by  GE/AES  s 
avionics  product  data  base  from  the  COs.  This 
plot  relates  systems  mean  time  between  failure  to 
aystem  complexity  as  determined  by  the  Navy  from 
airborne  avionics  experience  through  the  early 
1960s, 
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Figure  2.  Product  Requirements  and  Experience 
-  Thu  10M)it  and  1960s 

Thu  OE/AES  data  also  encompasses  airborne 
equipments  of  a  conventional  uouroiiundant  design, 
reflecting  nearly  100K  hours  of  equipment  levnl 
testing  on  a  diverse  complement  of  products  de¬ 
veloped  and  volume  produced  to  ruiinbll.ly  re¬ 
quirement*  for  a  number  of  government  and  in¬ 
dustrial  customers  ihroughout  the  decade.  Evi¬ 
dent  from  this  chart  is  a  trend  indicating  that 
through  the  decade  the  reliability  requirements 
and  product  complexity  have,  on  the  uverag.j, 
each  increased  by  n  factor  of  10. 

Projecting  forward  to  the  requirements  and 
product  needs  of  the  technically  sophisticated 70k, 
it  appears  reasonable  lo  expect  a  continuing  ex¬ 
tension  of  this  reliability  trend,  resulting  m  an 
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Additional  one  order  of  magnitude  Increase  In  re¬ 
quirements  paralleling  the  growth  of  the  1960s, 
While  ll  is  not  clear  today  exactly  the  form  this 
attention  will  take  in  the  1970s,  it  is  clear  that 
either  more  complex  equipments  with  today's 
mean  times  or  equipments  centering  at  the  cur¬ 
rent  high  end  of  the  complexity  scAle  with  more 
timely  achieved  reliability  will  be  required.  Rec¬ 
ognizing  the  difficulty  and  obvloua  lack  of  uniform 
success  experienced  In  coping  with  the  reliability 
requirements  of  the  1960s,  and  currently  project¬ 
ing  Increasing  requirement*  for  the  1970a,  It  la 
clear  that  a  change  in  the  practices  of  Reliability 
Planning  and  Management  le  mandatory.  To  meet 
this  need,  a  methodology  Is  required  which  las 
the  credibility  of  an  exacting  science,  incorpor¬ 
ates  the  flexibility  lor  alternative  program  plan¬ 
ning,  and  contains  standardisation  fnctora  to  fa¬ 
cilitate  a  uniform  evaluation  structure  suitable 
for  application  by  both  contractor  and  buyer. 

DEPARTURE  FROM  THE  CLASSIC  SCHEME 

The  classic  method  of  reliability  projection 
Is  prediction.  From  a  planning  viewpoint,  this 
typically  consists  of  a  preliminary  "before  de¬ 
sign"  and  a  refined  "during  design"  estimate  of 
the  product's  analytical  mean  time.  Classically 
and  erroneously,  buyer  and  seller  alike  assume 
that  this  analytical  prediction  is  representative 
of  the  expected  performance  of  the  product  ai  in¬ 
itially  manufactured.  This  is  not  the  case  in  the 
design  and  manufacture  of  avionics  products. 

Many  unknowns  and  intangibles  do  er.ist  that  can¬ 
not  be  analytically  forecast,  foreseen,  or  con¬ 
trolled.  Living  in  this  environment,  it  then  be¬ 
comes  clear  that  extended  product  experience  and 
familiarity  In  theflntendod  use  environment  is 
necessary  to  identify  a  whole  family  of  hidden  de¬ 
fects  for  corrective  action,  and  corrective  action 
tast  validation.  The  crux  of  the  reliability  plan¬ 
ning  problem  is  the  lack  of  ability  to  dimension 
(or  a  now  product  the  extent  of  the  evaluation  test 
need.  This  evaluation  may  be  extensive  enough  to 
affect  total  program  assets,  resources,  time,  and 
facilities.  Historically,  provisions  for  this  need 
are  not  dimensioned  and  provided.  This  Is  not  the 
case  at  GE/AES  where  consideration  Is  giver,  to 
the  interrelated  program  requirements  through 
the  use  of  Reliability  Planning  and  Management 
(R°M). 

RPM  CONCEPT 

To  cope  with  the  reliability  planning  problem 
of  bridging  the  gup  between  stated  reliability  mean 
time  numbers  and  the  reality  of  program  struc¬ 
turing  required  to  effect  a  product  tnat  Is  techni¬ 
cally  and  timely  compliant,  we  have  developed  a 
methodology  called  Reliability  Planning  and  Mun- 
ngement  (HPM).  This  concept  reduces  the  plan¬ 


ning  and  resource  allocation  requirements  Into  a 
simple  quantized  objectively  usable  format. 

Figure  3  portrays  the  basic  criteria,  con- 
•trainU,  and  factors  which  must  be  met  when 
structuring  a  reliability  program  compliant  v/ith 
the  concept.  To  apply  this  methodology,  it  is 
first  mandatory  that  no  laws  of  physics  are  vio¬ 
lated  by  the  designer  and  that  beyond- the-state- 
of-the-art  requirements  are  not  imposed.  Oper¬ 
ating  within  this  framework,  the  criteria  to  be 
fulfilled  consist  of  prediction  versus  requirement, 
initial  product  capability  assessment,  reliability 
growth  rate,  product  experience  gained  through 
extended  environmental  exposure,  calendar  time, 
and  change  constraints.  Successful  program  im¬ 
plementation  requires  that  specific  compliance  bo 
achieved  for  each  criterion  aa  follows. 

Prediction  versus  Requirement  specifies  that 
prior  to  release,  the  design  be  simplified  and 
parts  stress  and  screening  levels  be  adjusted 
until  an  analytical  reliability  prediction  based 
on  technically  established  and  credible  fail¬ 
ure  rates  (use  MIL  Handbook  217A)  will  yield 
at  a  minimum  a  prediction  125%  of  the  re¬ 
quirement 

Product  Capability  requires  that  a  realistic 
appraisal  be  made  of  the  new  or  changed  de¬ 
sign,  recognizing  the  inevitability  of  flaws 
which  constrain  initial  performance  to  10%  of 
the  Inherent  analytical  capability. 

Reliability  Rate  of  Growth  projects  that  reli- 
aHilTTy’I m pro v em enFfo r  complex  equipment 
when  operating  In  their  Intended  use  environ¬ 
ment  Is  approximately  inversely  proportional 

•  UESION  ■  SIMPLIFY  UNTIL  MIL  HOIK  21?  PREOICHON  IS  W* Of  REQUIREMENT 

•  PROCESSING  •  ADJUST  SCREENING  LEVEL  TO  MEET  MW  REQUIREMENT 

•  PLAN  PROGRAM  •  RASED  UPON  DUANE  GROWTH  AND  RPM  CRITERIA 

•  PAPER  RELEASE  DESIGN  •  APPROXIMATELY  10%  OF  INHERENT  ANALYTICAL  CAPABILITY 


PROGRAM  PLANNING  OPTIONS  PROGRAM  CONSTRAINTS 

•  EQUIPMENTS  •»  CHANCE  LIMITATIONS  •  *  CONMOL'-RATtON  CUNTROt 

•  FACILITIES  •  STATE  OMNI  ART 

•  LEVEL  Of  EFFORT  •  DOllARS 

•  PERFORMANCE  •  TIME 


Figure  3.  Reliability  Planning 
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(c  the  square  root  of  the  cumulative  operating 
(test)  time,  and  that  (or  a  constant  level  of 
corrective  action  effort  and  timely  Implemen¬ 
tation,  reliability  growth  closely  approxi¬ 
mates  a  straight  line  In  log- log  scales  as  in¬ 
itially  postulated  In  a  reliability  growth  mod¬ 
el  In  1962  by  J.T.  Duane.  Limits  of  relia¬ 
bility  rate  of  growth  for  avionics  have  been 
empirically  derived  by  GE/AES  from  the  cu¬ 
mulative  data  of  J.T.  Duane  and  OE/AES. 
Growth  limits  are  estimated  as  a  maximum 
rate  of  approximately  .6,  which  we  have  not 
achieved,  an  established  and  experienced 
rate  of  ,  5  for  a  hard-hitting  aggressive  reli¬ 
ability  program  with  management  support 
spanning  all  functions  of  a  knowledgable  or¬ 
ganization.  A  minimum  rate  of  .  1  can  be 
expected  on  those  programs  where  no  real 
specific  consideration  Is  given  to  or  for  re¬ 
liability.  In  this  latter  case,  growth  is 
largely  due  to  the  need  to  effect  solution  for 
the  clearly  obvious  problem  Impacting  pro¬ 
duction  and  from  corrective  action  taken  as 
a  result  of  ussr  experience. 

Product  Evaluation  Exposure  etructurea  the 
test  is aluatloiftlme  required  to  effect  a  com¬ 
pliant  product,  based  on  a  epee  if  lc,  initial 
capability,  growth  rate,  and  requirement. 
With  the  exposure  hours  established  and  a 
valid  assumption  on  achievable  test  efficiency 
(we  use  200  equipment  exposure  hours  per 
calendar  month  for  new  complex  avionics), 
then  the  tradeoffs  in  program  planning  can 
be  objectively  made  by  contractor  and  buyer, 
encompassing  the  acceptability  of  the  Initial 
design,  its  design  margin,  number  of  equip¬ 
ments  to  be  placed  on  test,  facilities,  test 
time,  calendar  time,  and  ultimately  program 
coat. 

Ori£in_of  Reliability  Growth 

The  basic  concept  of  a  patterned  reliability 
growth  is  the  Ijasis  for  the  evaluation  portion  of 
the  RPM  model:  first  recognized  and  published  by 
J.T.  Duane  of  GE  Company's  Motor  and  Genera¬ 
tor  Department  in  1962.  His  analysis  of  test  and 
operational  data  for  programs  with  test  times  as 
htgh  us  6  million  hours  on  five  divergent  groups 
of  products  (two  hydro-mechanical  devices,  two 
complex  aircraft  generators,  and  a  Jet  engine) 
formulated  a  pattern  which  resulted  In  the  follow¬ 
ing  concept  (see  Figure  t): 

Reliability  improvement  of  complex  equip¬ 
ment  follows  a  mathematically  predictable 
pattern. 

Reliability  Improvement  Is  approximately 
inversely  proportional  to  the  square  root  of 
cumulative  operating  (test)  time. 


Figure  4.  Original  Duane  Data 


For  a  constant  level  of  corrective  action  ef¬ 
fort  and  Implementation,  reliability  growth 
closely  approximates  a  straight  line  on  a  log 
scale. 

This  pattern  hat  beer,  confirmed  to  be  applicable 
to  avionics  equipment  by  GE/AES  from  data  on 
four  separate  programs. 

Experience  Performance  and  Model  Confirmation 

The  development  of  the  RPM  model  occurred 
ae  an  outgrowth  of  studies  aimed  at  deriving  a 
meaningful  reliability  development  program  pro¬ 
jection  model  from  actual  development  experi¬ 
ence.  Two  of  s»veral  GE  AES  programs  have 
been  selected  and  analyzed  In  this  paper  for  the 
purpose  of  illustrating  model  conformance.  In  e- 
valuating  achieved  performance  versus  that  which 
could  have  been  predicted  to  occur,  it  will  be 
shown  that  excellent  correlation  with  the  Duane 
growth  forecast  and  the  higher  order  RPM  model 
does  exist. 

The  first  Selected  program.  Figure  5,  con¬ 
ducted  between  1961  and  1965,  called  for  an  e- 
quipment  development  as  part  of  an  airborne  fire 
control  system  modernization.  From  the  history 
of  this  program,  it  is  observed  that  the  new  e- 
quipment  ‘nitlally  performed  at  approximately 
10%  of  the  predicted  MTBF  and  achieved  a  relia¬ 
bility  growth  rate  (a)  of  .5  during  a  comprehen¬ 
sive  reliability  program  over  an  extended  period 
of  evaluation  test  with  product  change  flexibility. 
Conversely,  significant  reliability  growth  did  not 
continue  during  production  under  disciplined 
change  constraints.  The  reliability  requirement 
achieved  and  demonstrated  during  development 
was  shown  to  persist  in  production  across  209 
equipments  for  25,360  test  hours. 

The  sec  snd  program,  Figure  6.  a  complex 
radar  equip  nent,  five  times  as  complex  ps  the 
first  program  and  a  vintage  of  the  midale  60s, 


had  essentially  ;he  same  attributes  as  the  first 
program.  Excellent  model  correlation  was  again 
shown  by  initially  performing  at  10%  of  the  pre¬ 
dicted  MTBF,  experiencing  a  growth  rate  of  .8, 
under  a  comprehensive  reliability  improvement 
program  and  slow  growth  under  production  con¬ 
straints. 
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model.  The  model,  now  recognized,  confirmed, 
and  dimensioned,  was  first  contractually  applied 
in  forward  fit  to  an  equipment  development  aimed 
at  alternate  and  expanded  capabilities  for  a  cur¬ 
rently  new  equipment.  The  task  involved  a  20%. 
design  change  to  an  11K  part  system  where  the 
first  production  lot  was  required  to  demonstrate 
reliability  performance  equal  to  that  specified  for 
the  unmodified  equipment. 

The  third  program,  Figure  7,  displays  the 
RPM  plan  and  development  experience  resulting 
in  an  equipment  completely  compliant  In  first-lot 
production  demonstration.  You  may  note  that 
this  program  plan  reflected  a  certain  lack  of  con¬ 
fidence  tr  that  a  conservative  a  of  .375  was  uti¬ 
lized.  Subsequent  to  the  successful  utilization  of 
the  model,  two  additional  equipment  developments 
have  been  undertaken,  utilizing  an  w  of  .5. 
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Figure  5.  RPM  Model  -  Program  1 
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Figure  6.  RPM  Model  -  Program  2 


Correlation  with  the  RPM  model  in  these  two 
examples  was  accomplished  after  development 
was  completed,  and  net  through  a  priori  program 
planning  as  presented  in  this  paper.  In  each  case, 
reliability  improvement  occurred  as  a  result  of 
intensive,  well-directed  recovery  programs  Im¬ 
plemented  at  the  expense  of  program  interrup¬ 
tions  and  delays.  The  second  program,  in  mid¬ 
course,  applied  the  Duane  growth  concept  as  a 
progress  monitoring  tool,  contributing  signifi¬ 
cantly  in  insight  to  the  formulation  of  the  RPM 


Sample  Application 

In  order  to  illustrate  the  decision-making 
power  of  the  RPM  model  and  to  describe  the 
steps  in  the  methodology  that  a  contractor  will 
take,  let  us  consider  a  sample  application,  typi¬ 
cal  of  those  confronting  us  in  today's  military 
environment. 

A  customer  generates  a  requirement  (Table 
11)  and  RFQ  to  design,  develop,  and  deliver  a  new 
avionics  equipment  which  violates  no  laws  of 
physics,  is  expected  to  be  within  the  state  of  the 
art,  has  a  lead  time  of  36  months,  and  an  appor¬ 
tioned  MTBF  from  a  higher  level  system  require- 


TABLE  n.  SAMPLE  APPLICATION 
IN  inUIMMXT 

•  «w  '  '<i  win  sun  or  mt 

•  140  IA4BCT  ilW,'  MM0N1HI 

•  urn  no  how;  -  «(ANNS  syjm  aikxtiomo 

•  iNOSAAMIHMU-ITO-m 

•  TiSTIUMl'STO-TIUMlI 

•  FAC  I  AND  CONhQUIATION  CONWW.  -  l«  FMOUCTION  Mill 
INITIAl  CONWACTOO  FIAN 

•  COMFlttlTY 

•  tSTIMATI •  111.  aOHOW  HKOICTICN 

•  HIOH  iniAllllTY  NOOIAM 
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•  tl  MI  MAW  FUN 

•  IS  MONTH  MUON,  0  MONTH  MFC,  II  MONTH  (VALUATION. 

)  MONTH  FIOMKTION  WANSITION 

•  l MPI  (MENTATION  OPTIONS 

•  MININUMAIsrS 

-I  IQUI FAINT,  SPAMS,  N  MONTH  TVMUATlON 

•  COMPU ANT  TIMS 

-SRHIIFAINTS.  SPAMS,  IT  MONTH  rVAUJATION 

•  lUST  NISH 

-TIQIII  PAINTS,  SPAMS.  I  MONTH  SVAUIATION 

-OIOWTH  CWTINQCNCV  IS4,  II  MONTH  IVAUIATIM 

merit  of  150  hour*.  Tho  reliability  program  la  to 
be  In  accordance  with  MIL-STD-786  with  teeting 
per  MIL-STD-781  teat  level  F;  First  Article  Con¬ 
figuration  Inspection  (FACI)  and  configuration  con¬ 
trol  are  required  on  the  first  production  item. 

Viewing  these  requirements,  a  contractor 
(tret  determines  a  functional  implementation  plan 
and  equipment  schedule  based  on  the  technical  ex¬ 
hibits,  previous  exparlsnce,  equipment  complexi¬ 
ty,  and  program  planning  judgments.  Let  ua  ao- 
sume  the  case  o:l  a  responsible  contractor  who  as¬ 
sesses  the  requirement  aa  being  within  the  state 
of  the  art  and  represents  an  equipment  of  UK 
parts  complexity.  Using  MIL-STD-217A  ns  a  de¬ 
parture  point,  a  prediction  of  220  hours  is  calcu¬ 
lated,  made  possible  by  use  of  screened  parts,  ap¬ 
plied  under  exacting  application  and  derating  cri¬ 
teria.  This  prediction  meet*  the  ftret  RPM  cri¬ 
teria  by  exceeding  the  minimum  125%  of  epeclfled 
requirement. 

The  schedule  milestones  are  established, 
based  on  past  development  experience,  resulting 
in  15  months  for  design,  6  additional  months  for 
initial  hardware  manufacture  and  ambient  test,  12 
months  for  (.valuation  testing,  and  3  months  for 
final  change  documentation  and  incorporation  pri¬ 
or  to  production  FACI  and  configuration  control 
constraints. 

The  second  and  third  criteria  of  the  model 
state  that  for  a  new  design  the  initial  hardware 
MTBF  win  be  10%  of  that  predicted,  and  reliabil¬ 
ity  growth  will  follow  the  Duane  postulate.  Em¬ 
ploying  these  criteria,  a  reliability  initial  esti¬ 
mate  and  growth  requirement  based  on  the  speci¬ 
fics  of  the  selected  implementation  now  can  be 
structured.  The  Initial  capability  for  this  new 
system  is  thus  dimensioned  as  22  hours,  10%  of 
the  predicted  MTBF,  A  growth  rate  a  of  .6  is 
planned  based  upon  a  comprehensive  reliability 
program  executed  through  competent  implemen¬ 


tation  of  M1L-8TD-785  and  M1L-STD-781.  The 
growth  requirements,  Figure  8,  Indicate  that 
compliance  can  be  achieved  at  4800  hours  of  test 
time. 
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Figure  8.  Planning  Example 

In  implementation  planning,  optimization  ol 
the  reliability  program  about  individual  program 
priorities  is  desirable  and  required.  In  this  ex¬ 
ample,  the  contractor  elects  to  consider  optimi¬ 
zation  in  three  cases  of  program  structuring; 
minimum  assets,  compliant  time,  and  least  risk. 
The  objective  is  clearly  to  dimension  and  evaluate 
the  alternatives  and  activity  necessary  to  achieve 
roqulred  equipment  reliability  during  development 
and  prior  to  the  production  and  user  constraints. 
Based  on  a  review  of  test  experience  with  com¬ 
plex  systems,  it  has  been  established  for  GE/AES 
products  that  200  hours  of  test  operate  time  can 
reasonably  be  achieved  per  system  month  of 
effort. 

Operating  within  these  bounds,  let  us  now 
consider  the  test  plan  options  open  to  the  contrac¬ 
tor  to  develop  a  compliant  product.  The  first  op¬ 
tion,  minimum  assets,  requires  one  system  tested 
continuously  for  24  months  representing  the  least 
number  of  systems.  The  second  option,  a  time 
compliant  test,  requires  two  systems  tested  con¬ 
tinuously  and  concurrently  tor  12  months.  The 
third  option,  least  risk,  requires  three  systems 
tested  continuously  and  concurrently  for  8  months, 
accommodating  additional  time  for  reaction  to 
contingency  including  growth  of  up  to  25%  in  prod¬ 
uct  complexity.  This  growth  reflects  the  case 
where  an  11K  part  original  estimate  grows  during 
detail  implementation  to  a  14K  part  system. 

From  these  optimizations,  the  magnitude  of 
the  program,  kinds  of  disciplines,  and  avails'. le 
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tradeoffs  bour.  ting  a  successful  program  are  now 
clear  (o  management. 


CONCLUSION 

In  conclusion,  RPM  is  essentially  a  manage¬ 
ment  tool  for  bridging  the  gap  betwssn  stated  reli¬ 
ability  requirements  and  implementation  planning. 
The  RPM  methodology,  equally  usable  by  buyer 


and  contractor,  la  applicable  to  establishing  plans, 
projecting  effort,  evaluating  proposals  ami  mon¬ 
itoring  contract  performance.  Experience  at 
OE/aES  haa  ehown  that  the  RPM  methodology  in 
practice  doee  addreee  and  effectively  contribute 
to  aolving  the  reliability  planning  problem  a*  dt- 
mena toned  in  this  paper.  Thle  Initial  RPM  pre¬ 
sentation  will  hopefully  foeter  amplification  and 
refinement  of  the  model  to  the  point  of  a  uniform 
induetry  accepted  and  practiced  methodology. 
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This  report  presents  the  results  of  a  study  to  develop  basic  relationahipB 
capable  of  determining  and  predicting  the  reliability  oostB  attributable  to 
equipment  reliability.  The  study  is  based  on  data  derived  during  the  develop- 
ment  phase  of  equipment  acquisition.  The  relationships,  including  increments.: 
reliability  gain  related  to  incremental  reliability  cost,  were  developed  using 
data  from  two  manufacturers  on  ten  equipments  covering  three  reliability  ele¬ 
ments.  Thu  equipments  cover  a  span  of  end  use  airborne  and  space  environments . 
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20.  ABSTRACT  (continued) 

The  reliability  elements  analyzed  in  the  study  are  the  reliability  design  pro¬ 
gram  (includes  prediction,  failure  mode  and  effect  analysis,  and  design  re¬ 
views);  the  reliability  parts  program  (includes  parts  screening  specification, 
parts  standardisation  and  control,  and  vendor  control);  and  the  reliability 
testing  program  (includes  evaluation  testing,  equipment  environmental  screen¬ 
ing,  and  reliability  demonstration  testing).  To  develop  the  relationships, 
tvo  linear  models  were  hypothesized.  The  first  model  relates  resultant  equip¬ 
ment  reliability,  the  reliability  costa  and  the  equipment  complexity.  The 
second  model  relates  incremental  reliability  gain  to  reliability' element  ooat. 
The  initial  MTBF  (equipment  reliability  prior  to  the  start  of  a  reliability 
program)  is  defined  in  terms  of  the  normalized  equipment  complexity.  Detailed 
discussions  of  data  collection  and  analysis  efforts  along  with  step-by-step 
procedures  for  using  the  modeling  results  are  presented  in  the  report.-  Con-  ■ 
stralnts  and  precautions  to  be  applied  in  using  the  equational  relationships 
ore  also  Included. 
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